feat: data pipeline changes for segmentRepositories [IN-552] [IN-554] #3326

borfast · 2025-08-12T22:29:20Z

In the context of IN-524, we want to have the archived and excluded repositories for each project returned along with the project data.

This means adding the repository data to TinyBird, which already includes the archived and excluded flags in the segmentRepositories table, and then merge that data into the existing pipes that return project data, namely the projects_list and the search_collections_projects_repos pipes.

This PR:

Adds the database migration to ensure we get data from the segmentRepositories table into Sequin.
Adds a new segmentRepositories data source in TinyBird to receive data from the aforementioned table.
Modifies the insights_projects_populated_copy pipe to include the list of archived and excluded repositories, which is then propagated to the insights_projects_populated_ds data source, from there it feeds the insightsProjects_filtered pipe, and finally the projects_list pipe, which is the end goal.
- The search_collections_projects_repos contains archived and excluded columns for the repositories results. This required adding a JOIN to the query, and with all the optimisations that were done not too long ago, I'm not sure if this

Specific tickets in Jira:

github-actions

Conventional Commits FTW!

Copilot

Pull Request Overview

This PR implements the infrastructure to propagate archived and excluded repository data from the segmentRepositories table through the TinyBird pipeline to support enhanced project filtering and search capabilities.

Adds database migration to enable replication of the segmentRepositories table to TinyBird
Creates a new segmentRepositories data source in TinyBird and updates pipeline to aggregate archived/excluded repository lists
Propagates archived repositories data through the insights project pipeline to enable repository-aware search functionality

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
backend/src/database/migrations/V1753798345__segmentRepositories_replication.sql	Enables replication of segmentRepositories table to TinyBird
services/libs/tinybird/datasources/segmentRepositories.datasource	Defines TinyBird data source schema for segmentRepositories table
services/libs/tinybird/datasources/insights_projects_populated_ds.datasource	Adds archived/excluded repositories fields to project data source
services/libs/tinybird/pipes/insights_projects_populated_copy.pipe	Aggregates archived/excluded repositories and joins to project data
services/libs/tinybird/pipes/insightsProjects_filtered.pipe	Passes through archived/excluded repositories to filtered results
services/libs/tinybird/pipes/search_collections_projects_repos.pipe	Exposes archived repositories in search results

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

services/libs/tinybird/pipes/insights_projects_populated_copy.pipe

backend/src/database/migrations/V1753798345__segmentRepositories_replication.sql

services/libs/tinybird/datasources/insights_projects_populated_ds.datasource

services/libs/tinybird/datasources/segmentRepositories.datasource

borfast · 2025-08-27T14:16:43Z

Ping @epipav

joanagmaia · 2025-08-28T10:34:41Z

@borfast can you confirm if the PR is up to date with the latest comments? And if so, can you gather all the questions you need to align with the team and perhaps go over them on a quick sync

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

…olumns Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

epipav

I think the only issue is the ENGINE_VER of the new datasource, other than that looks good 👍

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

borfast requested a review from Copilot August 12, 2025 22:29

borfast self-assigned this Aug 12, 2025

borfast requested a review from epipav as a code owner August 12, 2025 22:29

github-actions bot reviewed Aug 12, 2025

View reviewed changes

Copilot AI reviewed Aug 12, 2025

View reviewed changes

services/libs/tinybird/pipes/insights_projects_populated_copy.pipe Outdated Show resolved Hide resolved

borfast changed the title ~~Sata pipeline changes for segmentRepositories [IN-552]~~ feat: data pipeline changes for segmentRepositories [IN-552] [IN-554] Aug 12, 2025

epipav requested changes Aug 15, 2025

View reviewed changes

borfast requested a review from themarolt as a code owner August 16, 2025 11:31

borfast force-pushed the feature/in-524-data-pipeline-changes-for-segmentRepositories branch from eea8f5c to 4501d3e Compare August 28, 2025 00:31

borfast requested a review from epipav August 28, 2025 00:35

borfast force-pushed the feature/in-524-data-pipeline-changes-for-segmentRepositories branch from 4501d3e to 3a3d9af Compare August 28, 2025 15:43

borfast added 10 commits August 30, 2025 00:55

feat: replication changes for the segmentRepositories table

7ad46f9

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

feat: create the segmentRepositories data source in TinyBird

0fb2a8c

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

feat: add the archived and excluded columns to the projects in TinyBird

f1ae4c6

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

feat: add the archived project column to the search endpoint in TinyBird

bc183d9

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

chore: fix node name

307cd45

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

chore: change archived and excluded flags in the search pipe

e6b2939

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

chore: add updatedAt column to segmentRepositories

3666ea4

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

chore: clarification about the purpose of the archived and excluded c…

f6e780e

…olumns Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

chore: add updatedAt column to segmentRepositories

3b1bdbf

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

chore: fix regression in column name after rebase

f7f5eb7

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

borfast force-pushed the feature/in-524-data-pipeline-changes-for-segmentRepositories branch from e78438a to f7f5eb7 Compare August 29, 2025 23:56

epipav requested changes Sep 1, 2025

View reviewed changes

borfast added 5 commits September 1, 2025 16:19

fix: change segmentRepositories.datasource ENGINE_VER to updatedAt

5c3d585

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

chore: update pnpm

2ff7ee8

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

fix: fix parsing some Gitlab repository URLs

428ef45

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

fix: fix offset when fetching repositories

d5f6fd1

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

chore: mark 404 and 403 as excluded

3f852a5

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

borfast added 2 commits September 1, 2025 16:21

fix: fix SQL error

21a292d

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

chore: recheck repositories every 3 days, not 5

5aa4c04

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

borfast added the --skip-regression-tests To skip regression tests in Tinybird CI label Sep 1, 2025

borfast requested a review from epipav September 1, 2025 15:30

epipav approved these changes Sep 1, 2025

View reviewed changes

borfast removed the --skip-regression-tests To skip regression tests in Tinybird CI label Sep 1, 2025

borfast added 4 commits September 1, 2025 18:14

fix: fix segmentRepositories.datasource formatting

b8fa46d

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

fix: fix insights_projects_populated_copy.pipe formatting

4008d0e

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

fix: fix insightsProjects_filtered.pipe SQL

5a6bd81

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

fix: fix insights_projects_populated_copy.pipe SQL

1d7cd18

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

borfast force-pushed the feature/in-524-data-pipeline-changes-for-segmentRepositories branch from 83c54c6 to 1d7cd18 Compare September 1, 2025 17:15

fix: fix search_collections_projects_repos.pipe SQL

0c2b86c

Signed-off-by: Raúl Santos <4837+borfast@users.noreply.github.com>

borfast merged commit 1dde2d5 into main Sep 1, 2025
14 of 15 checks passed

borfast deleted the feature/in-524-data-pipeline-changes-for-segmentRepositories branch September 1, 2025 17:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: data pipeline changes for segmentRepositories [IN-552] [IN-554] #3326

feat: data pipeline changes for segmentRepositories [IN-552] [IN-554] #3326

borfast commented Aug 12, 2025 •

edited

Loading

Uh oh!

github-actions bot left a comment •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

borfast commented Aug 27, 2025

Uh oh!

joanagmaia commented Aug 28, 2025

Uh oh!

epipav left a comment

Uh oh!

Uh oh!

Uh oh!

feat: data pipeline changes for segmentRepositories [IN-552] [IN-554] #3326

feat: data pipeline changes for segmentRepositories [IN-552] [IN-554] #3326

Conversation

borfast commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

borfast commented Aug 27, 2025

Uh oh!

joanagmaia commented Aug 28, 2025

Uh oh!

epipav left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

borfast commented Aug 12, 2025 •

edited

Loading

github-actions bot left a comment •

edited

Loading